{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "## Classification - Adult Census\n",
    "\n",
    "In this example, we try to predict incomes from the *Adult Census* dataset.\n",
    "\n",
    "First, we import the packages (use `help(synapse)` to view contents),"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from pyspark.sql import SparkSession\n",
    "\n",
    "# Bootstrap Spark Session\n",
    "spark = SparkSession.builder.getOrCreate()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now let's read the data and split it to train and test sets:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "data = spark.read.parquet(\n",
    "    \"wasbs://publicwasb@mmlspark.blob.core.windows.net/AdultCensusIncome.parquet\"\n",
    ")\n",
    "data = data.select([\"education\", \"marital-status\", \"hours-per-week\", \"income\"])\n",
    "train, test = data.randomSplit([0.75, 0.25], seed=123)\n",
    "train.limit(10).toPandas()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "`TrainClassifier` can be used to initialize and fit a model, it wraps SparkML classifiers.\n",
    "You can use `help(synapse.ml.train.TrainClassifier)` to view the different parameters.\n",
    "\n",
    "Note that it implicitly converts the data into the format expected by the algorithm: tokenize\n",
    "and hash strings, one-hot encodes categorical variables, assembles the features into a vector\n",
    "and so on.  The parameter `numFeatures` controls the number of hashed features."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from synapse.ml.train import TrainClassifier\n",
    "from pyspark.ml.classification import LogisticRegression\n",
    "\n",
    "model = TrainClassifier(\n",
    "    model=LogisticRegression(), labelCol=\"income\", numFeatures=256\n",
    ").fit(train)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finally, we save the model so it can be used in a scoring program."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "from synapse.ml.core.platform import *\n",
    "\n",
    "if running_on_synapse():\n",
    "    model.write().overwrite().save(\n",
    "        \"abfss://synapse@mmlsparkeuap.dfs.core.windows.net/models/AdultCensus.mml\"\n",
    "    )\n",
    "elif running_on_databricks():\n",
    "    model.write().overwrite().save(\"dbfs:/AdultCensus.mml\")\n",
    "elif running_on_binder():\n",
    "    model.write().overwrite().save(\"/tmp/AdultCensus.mml\")\n",
    "else:\n",
    "    print(f\"{current_platform()} platform not supported\")"
   ],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
